Papers by Hal Daumé III
Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain question answering uses evidence retrieved from large corpus to answer questions . state-of-the-art approaches require intermediate evidence annotations for training . however, such intermediate annotations are expensive and methods that rely on them cannot transfer to the more common setting . |
| Approach: | They propose an open-domain question answering approach that alternately finds evidence from an up-to-date model and encourages the model to learn the most likely evidence. |
| Outcome: | The proposed approach improves over weak retrievers on multi-hop and single-hop benchmarks without using evidence labels. |
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models encode correlations between social groups and traits, like associating the group with the group. |
| Approach: | They adapt the Agency-Belief-Communion (ABC) stereotype model to a language model and introduce the sensitivity test (SeT) to measure stereotypical associations. |
| Outcome: | The proposed framework is used to measure stereotyping of intersectional identities in language models. |
Analyzing Stereotypes in Generative Text Inference Tasks (2021.findings-acl)
Copied to clipboard
| Challenge: | Social psychology studies how social stereotypes are shared as part of cultural knowledge . |
| Approach: | They study how stereotypes manifest when potential targets are situated in neutral contexts . they collect human judgments on the presence of stereotypes in generated inferences based on annotator positionality . |
| Outcome: | The results show that the annotators' positions differ depending on the type of inferences they generate . |
What’s Different between Visual Question Answering for Machine “Understanding” Versus for Accessibility? (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing benchmarking datasets for visual question answering focus on machine "understanding" but it remains unclear how progress on those datasets corresponds to improvements in this real-world use case. |
| Approach: | They evaluate the visual question answering task by evaluating a variety of VQA models. |
| Outcome: | The proposed model can achieve high scores on tasks thought to require human-like comprehension, including image tagging and captioning. |
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022.naacl-main)
Copied to clipboard
| Challenge: | Evaluating natural language generation systems is difficult, as there are many ways to express similar things in text. |
| Approach: | They combine interviews with NLG practitioners to examine ethical considerations and their implications for NLG evaluation. |
| Outcome: | The findings of the study surface goals, community practices, assumptions, and constraints that shape NLG evaluations, and examine their implications and how they embody ethical considerations. |
FairPrism: Evaluating Fairness-Related Harms in Text Generation (2023.acl-long)
Copied to clipboard
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
| Challenge: | FairPrism dataset provides a framework for measuring and mitigating fairness-related harms caused by AI text generation systems. |
| Approach: | They propose a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. |
| Outcome: | FairPrism is a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering harms relating to gender and sexuality. |
Hallucination Detection for Grounded Instruction Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for generating instructions for navigation generate references to objects or actions that are inconsistent with what a human follower would perform or encounter along the path. |
| Approach: | They propose a weakly supervised approach that detects hallucinated references by using a pre-trained vision-language model. |
| Outcome: | The proposed model outperforms baseline models and supervised models on generating navigation instructions. |
Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree (2023.findings-acl)
Copied to clipboard
| Challenge: | Disagreement in annotations is natural for humans, depending on background, identity, positionality . many active learning approaches focus on examples where model entropy and annotator entropicy are the most different. |
| Approach: | They propose an active learning approach that focuses annotations on examples where model entropy and annotator entropic are the most different. |
| Outcome: | The proposed approach reduces the number of annotations required by 24% on average across datasets. |
Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventions (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that model steering can preserve fluency and unrelated abilities, but it fails to preserve robustness specificity. |
| Approach: | They propose a framework that distinguishes three dimensions of specificity: general, control, and robustness. |
| Outcome: | The proposed framework distinguishes three dimensions of specificity: general (preserving fluency and unrelated abilities), control (preserving related control properties), and robustness (preserving control properties under distribution shifts). |
Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Current methods for complex question answering use structured knowledge and unstructured text. |
| Approach: | They propose a multi-step retrieval approach that iteratively forms an evidence chain through beam search in dense representations. |
| Outcome: | The proposed method is competitive to state-of-the-art systems without using semi-structured information. |
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (D19-1)
Copied to clipboard
| Challenge: | HANNA is an interactive photo-realistic simulator that can help agents with navigation tasks . human assistants are rich external knowledge sources but may not be available all the time to provide guidance . |
| Approach: | They develop a photo-realistic mobile agent simulator that asks for help from humans . they use natural language and visual instructions to direct agents towards the goals . |
| Outcome: | The proposed approach can accomplish tasks more effectively than competing models . it can also predict its own chances of making future progress . |
Towards Conceptualization of “Fair Explanation”: Disparate Impacts of anti-Asian Hate Speech Explanations on Content Moderators (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work at the intersection of AI explainability and fairness has focused on how explanations can improve human-plus-AI task performance . |
| Approach: | They propose to characterize what constitutes an explanation that is itself "fair" they use not just accuracy and label time, but psychological impact of explanations on different groups . |
| Outcome: | The proposed method is based on content moderation of potential hate speech and its differential impact on Asian vs. non-Asian proxy moderators across explanation approaches. |
Answer-based Adversarial Training for Generating Clarification Questions (N19-1)
Copied to clipboard
| Challenge: | a goal of natural language processing is to develop techniques that enable machines to process naturally occurring language. |
| Approach: | They propose a model where hypothetical answers are latent variables that can guide the model into generating more useful clarification questions. |
| Outcome: | The proposed model outperforms retrieval-based models and ablations that exclude utility model and adversarial training on two datasets. |
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020.acl-main)
Copied to clipboard
| Challenge: | 146 papers analyzing "bias" in NLP systems lack normative reasoning, we find . authors propose three recommendations for work analyzing “bias” in Nlp systems . |
| Approach: | They propose three recommendations for analyzing "bias" in NLP systems . they propose to focus on what kinds of system behaviors are harmful, in what ways, to whom, and why . |
| Outcome: | The proposed methods for measuring or mitigating “bias” are poorly matched to their motivations and do not engage critically with literature outside of NLP. |
Factual or Contextual? Disentangling Error Types in Entity Description Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation practices only distinguish between model generated referring expressions being accurate (ground-truth) versus inaccurate (not groundtruth). |
| Approach: | They propose to integrate indicators for factual inconsistencies and contextual incongruities into automated evaluations of language models to assess the differences in error types across familiar vs unfamiliar entities. |
| Outcome: | The proposed evaluation paradigm disentangles factuality and congruity errors in natural contexts. |
Global Voices: Crossing Borders in Automatic News Summarization (D19-54)
Copied to clipboard
| Challenge: | a crowd-sourced dataset is needed to evaluate cross-lingual summarization methods . human-written summarizing is expensive and difficult to design for humans . |
| Approach: | They construct a multilingual dataset for evaluating cross-lingual summarization methods . they use social-network descriptions of news articles to extract evaluation data . |
| Outcome: | The proposed dataset compares a translate-then-summarize approach with baselines in 15 languages. |
Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work examines the cognitive capabilities of language models through psychological tests designed for humans. |
| Approach: | They propose to use human-like cognitive capabilities to evaluate language models . they propose to augment language models with better listeners to improve their performance . |
| Outcome: | The proposed method boosts language models with better models of the listener and improves them. |
Heterogeneous Supervised Topic Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Researchers in the social sciences are interested in the relationship between text and an outcome of interest. |
| Approach: | They develop a probabilistic approach to text analysis and prediction using a joint model of text and outcomes to find heterogeneous patterns. |
| Outcome: | The proposed model outperforms other methods on eight datasets and consistently outperformed other models. |
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information (P18-1)
Copied to clipboard
| Challenge: | StackExchange users routinely ask clarifying questions to fill information gaps . a principle goal of asking questions is to fill this information gap . |
| Approach: | They build a model to rank candidates by their usefulness to a given post . they use data from StackExchange to evaluate the model against human judgments . |
| Outcome: | The proposed model outperforms baselines on 500 samples of StackExchange's clarification questions. |
Active Imitation Learning with Noisy Guidance (2020.acl-main)
Copied to clipboard
| Challenge: | Structured prediction methods learn models to map inputs to complex outputs with internal dependencies. |
| Approach: | They propose an algorithm that mimics an expert's choice at any queried state . they apply LEAQI to three sequence labelling tasks to reduce query costs . |
| Outcome: | The proposed algorithm shows better accuracies over a passive approach. |
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale semantic parsing datasets annotated with logical forms have enabled advances in supervised approaches. |
| Approach: | They propose to enrich English-language questions with SQL equivalents and alignments . they propose to use supervised attention and an auxiliary objective to disambiguate references . |
| Outcome: | The proposed method improves over strong baselines by 4.4% execution accuracy. |
Content Selection in Deep Learning Models of Summarization (D18-1)
Copied to clipboard
| Challenge: | Using deep learning models, we find that word embedding does not improve performance over simpler models. |
| Approach: | They propose to use sentence embedding to perform content selection across multiple domains . they propose to propose two alternative models that use auto-regressive sentence extraction . |
| Outcome: | The proposed models improve performance across news, personal stories, meetings, and medical articles. |
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems (2023.emnlp-main)
Copied to clipboard
Navita Goyal, Eleftheria Briakou, Amanda Liu, Connor Baumler, Claire Bonial, Jeffrey Micher, Clare Voss, Marine Carpuat, Hal Daumé III
| Challenge: | Existing NLP systems can only access the retrieved context to determine the answer, resulting in a knowledge gap between the information that is required to answer the question and the information available to assess the model’s correctness. |
| Approach: | They ask whether adding relevant background helps mitigate users’ over-reliance on predictions. |
| Outcome: | The proposed approach reduces over-reliance on model predictions even in the absence of sufficient information to assess their correctness. |
Comparing and Developing Tools to Measure the Readability of Domain-Specific Texts (D19-1)
Copied to clipboard
Elissa Redmiles, Lisa Maszkiewicz, Emily Hwang, Dhruv Kuchhal, Everest Liu, Miraida Morales, Denis Peskov, Sudha Rao, Rock Stevens, Kristina Gligorić, Sean Kross, Michelle Mazurek, Hal Daumé III
| Challenge: | Despite this, we lack a thorough understanding of how to validly measure readability at scale, especially for domain-specific texts. |
| Approach: | They present a comparison of the validity of well-known readability measures and introduce a novel approach to measure readability at scale. |
| Outcome: | The proposed approach addresses shortcomings of existing measures. |
It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance (2023.findings-acl)
Copied to clipboard
| Challenge: | a meta-analysis and survey of practitioners reveal that benchmarks suffer from operationalization disagreements. |
| Approach: | They propose a taxonomy of disagreement to explain disagreements in NLP benchmarks . they propose defining how tasks are conceptualized and operationalizing benchmarks to document their limitations. |
| Outcome: | The proposed taxonomy identifies two types of disagreements among NLP practitioners . it shows that benchmarks are not clearly conceptualized and suffer from operationalization disagreements . |
A Rose by Any Other Name would not Smell as Sweet: Social Bias in Names Mistranslation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using demographics, we hypothesize that the ability of translation systems to correctly translate female-associated names is significantly lower than male-associated name. |
| Approach: | They propose a translation evaluation procedure based on round-trip translation of names that are demographically aligned and analyze the effect of name demographics on translation quality using generalized linear mixed effects models. |
| Outcome: | The proposed evaluation procedure is based on round-trip translation of names from a dataset of names that are demographically aligned and shows that the ability of translation systems to translate female-associated names is significantly lower than male-associated name. |
Toward Gender-Inclusive Coreference Resolution (2020.acl-main)
Copied to clipboard
| Challenge: | a recent study shows that coreference resolution systems can be harmful to binary and non-binary trans and cis stakeholders. |
| Approach: | They propose to use gender-based crowd annotations to investigate coreference resolution biases . they use a dataset to examine the complexity of gender in crowd annotation systems . |
| Outcome: | a new study shows that without acknowledging and building systems that recognize gender, we build systems that lead to many potential harms. |